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Abstract Body 


Background / Context: 

Studies from K-12 curricular research (O’Donnell, 2008) as well as school-based 
prevention (Greenberg, Domitrovich, Graczyk, & Zins, 2005) suggest that curricula implemented 
without a high degree of fidelity will fail to produce intended benefits. The importance of 
examining implementation quality during the evaluation of interventions has been recognized for 
decades (Carroll et al., 2007; Moncher & Prinz, 1991; Sechrest, West, Phillips, Redner, & 
Yeaton, 1979), yet there tend to be strikingly high rates of variability in implementation for past 
and recent trials of school-based program effectiveness (Berman & McLaughlin, 1976; 

Rohrbach, Grana, Sussman, & Valente, 2006). This is particularly important given Domitrovich 
and Greenberg’s (2000) review of several social-emotional learning (SEL) programs, which 
found that implementation fidelity was closely linked to positive program outcomes for teachers 
and students. Thus, Chen (1998) and others (Devaney, O’Brien, Resnik, Keister, & Weissberg, 
2006) suggest that the system in place for implementing interventions may be as important as the 
intervention itself in contributing to program outcomes. 

Teacher-educators and policy-makers recognize that ongoing training and support for 
high quality implementation of curricula can be a vital component of systems that ensure the 
value of education experiences, particularly for students at-risk of school failure (Meisels, 2007; 
Pew Charitable Trusts, 2007; Pianta, 2005). In particular, there is growing consensus that one- 
on-one consultation approaches (also referred to as coaching, mentoring, staff development) that 
provide ongoing support and feedback may be the most direct, effective path to producing high 
quality implementation of curricula (Ingersoll & Kralik, 2004; Landry et al., 2006; Pianta & 
Allen, 2009). However, there is as much potential for variability in the quality of coaching as 
there is in a teacher’s fidelity to a curriculum, which raises several questions about how these 
different levels of intervention fidelity are related to one another. 

Purpose / Objective / Research Question / Focus of Study: 

The current study is embedded within the evaluation of an integrated SEL and literacy 
program in third through fifth grade classrooms called the 4Rs (Reading, Writing, Respect, and 
Resolution; Morningside Center for Teaching Social Responsibility, 2001). A recent randomized 
controlled evaluation of 4Rs indicated moderate to large, positive impacts on teacher practices 
and small, positive impacts on student outcomes (Brown, Jones, LaRusso & Aber, 2010; Jones, 
Brown & Aber, 2011). Despite the promise of these findings, there was considerable variability 
during implementation of the 4Rs Program in curriculum delivery, which may have attenuated 
intervention effects. Therefore, in the current study teachers using the 4Rs curriculum are 
supported with a standardized, video-based coaching program called MyTeachingPartner (MTP; 
Pianta et al., 2008). In past trials with a variety of curricula across prek and secondary 
classrooms, MTP yielded improvements on teacher implementation and teaching practices that 
were moderate to large (Pianta et al., 2008). 

As part of this evaluation of 4Rs+MTP, we are closely and objectively monitoring the 
quality with which coaches implement the 2 main components of MTP: (1) on-line written 
prompts that ask teachers to analyze key interactions from their own videotaped 4Rs lessons, and 
(2) in-person conferences that deepen teachers analysis of their 4Rs practice and result in 
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actionable improvement plans. In addition, videotaped footage of teachers’ implementation of 
4Rs are coded by independent observers for fidelity to the curriculum protocol. These data allow 
us to explore the following questions about patterns of variability in these 2 levels of 
implementation and how they relate to one another: 

1. To what extent are the two measures of coaches’ quality of implementation (prompt- 
writing, conferences) related to one another? 

2. To what extent is there variability on these two measures within coaches, across their 
caseload of teachers? 

3. To what extent does quality of coaching predict variation in teachers’ implementation of 
the 4Rs curriculum, above and beyond a global measure of teaching quality (CLASS)? 

Setting: 

The 4Rs+MTP intervention is being implemented in 6 New York City public elementary 
schools. 

Population / Participants / Subjects: 

Across the 6 schools, 35 teachers are participating from grades 3-5. This sample includes 
18 regular education classrooms, 3 special education classrooms, 10 inclusion classrooms (mix 
of regular and special education students) and 4 dual language classrooms. Schools and teachers 
are distributed between 2 4Rs coaches, with one coach serving 3 schools and 16 teachers and the 
other coach serving 4 schools and 19 teachers. 

Intervention / Program / Practice: 

The 4Rs program has two primary components: (1) a comprehensive 7-unit, 21 lesson, 
literacy-based curriculum in conflict resolution and social-emotional learning and (2) 25 hours of 
teacher training, plus ongoing coaching. The 4Rs curriculum uses high quality children’s 
literature as a springboard for helping students gain skills and understanding in handling anger, 
listening, assertiveness, cooperation, negotiation, mediation, and building community. Each unit 
begins with the teacher reading aloud a children’s book, carefully chosen to introduce the main 
theme of the unit. After the read-aloud is “Book Talk,” which includes discussion, writing, and 
role-playing aimed at deepening students’ understanding of the book. The rest of the unit 
consists of “Applied Learning” lessons in conflict resolution and social-emotional skills related 
to the theme. Lessons are taught every week for 30-45 minutes throughout the academic year. 

MTP is an innovative approach to supporting curriculum implementation that unites the 
ideas of providing ongoing, personalized feedback and support to teachers and embedding these 
implementation supports within a validated framework that emphasizes the importance of 
teacher- student interactions to ensure effectiveness of curricula. The premise of MTP is that 
professional development for teachers can improve the implementation of curricula through 
provision of extensive opportunities for individualized feedback and support for effectiveness in 
one’s own instruction, implementation, and interactions with students. Importantly, the 
Classroom Assessment Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008), a shared 
framework for defining and observing teacher-student interactions that has shown strong links to 
growth in student outcomes, guides all observations, feedback, and support. Every two weeks, 
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teachers videotape their implementation of instructional activities and send this footage to their 
coach. The coach then edits the tape into short segments that focus on a specific dimension of 
interaction, as defined by the CLASS (e.g., teacher sensitivity), and its application to 
implementation of the instructional activity. The short video segments and accompanying written 
feedback and questions (called prompts) are posted to each teacher’s private website. Teachers 
view the segments and respond to the accompanying prompts. Based on a long history of similar 
work focused on parent-child interactions (Dishion, Shaw, Connell, Gardner, Weaver, & Wilson, 
2008; Juffer, Bakermans-Kranenburg, & van IJzendoom, 2008; Mendelsohn et al., 2007; 
Webster-Stratton & Taylor, 2001), the intention of these prompts is to focus the teacher's 
attention on specific aspects of her behavior toward students and the students’ responses. 
Teachers then meet with their coach for an in-person 30-minute conference to discuss the 
prompts and feedback, the teachers’ responses, and to problem-solve and plan for future lessons. 
In this study, teachers will engage in 8 MTP coaching cycles throughout the academic year. 

Research Design: 

The study is a quasi-experimental, matched control group design. However, the current 
questions involve within-treatment group analyses. 

Data Collection and Analysis: 

Data on the quality of coaches’ online-prompts to teachers will be assessed based on the 
written record of coach prompts stored and accessible to our research team via each teacher’s 
individual website (3 prompts across 4 different coaching cycles = 12 prompts/teacher). The 
quality of each coaches online prompts to their teachers will be scored based on a coding rubric 
modified for use in this study. Eleven items, scored on a 3-pt. Likert scale (low quality=l, high 
quality=3), assess the coaches’ prompts with regard to their specificity and detail in focusing 
teachers’ attention on their interactions with students (e.g., CLASS dimension of focus is clearly 
defined; prompts are succinct, focused, and identify details about interactions with specific 
students) and the extent to which the coaches’ questions/statements push teachers to reflect on 
and analyze those interactions and to consider something new about their practice. 

Data on the quality of in-person conferences between coaches and teachers is being 
assessed using the transcripts of audiotaped coach-teacher conferences. The quality of 3-4 
coaching conferences for each teacher will be scored based on a rubric developed for use in this 
study. Seventeen items, scored on a 3-point Likert scale (low quality=l, high quality=3), assess 
the extent to which in-person conferences deepen teachers’ analysis of their 4Rs practice and 
result in actionable improvement plans. More specifically, items assess the degree to which 
coaches: achieve a balance between guiding the conference and promoting teacher expression, 
focus on specific teacher- student interactions, focus on the effects of teacher actions on students’ 
learning and behavior, facilitate the teacher’s ability to generate their own strategies for 
managing challenging situations, collaborate with teachers in identifying actionable next steps, 
and maintain a non-judgmental attitude towards the teacher’s actions. 

The quality of teachers’ implementation of 4Rs lessons will be assessed using videotaped 
footage teachers take of themselves implementing 4Rs lessons in each of the first four coaching 
cycles of the year. Each full lesson videotape will be scored by independent observers based on a 
rubric developed specifically for this study to assess fidelity to the curriculum protocol. 
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Specifically, using a 5-pt. Likert scale (l=low, 5=high) that includes clear behavioral anchors 
characterizing each response category, observers rate the quality with which teachers conduct the 
read aloud (e.g., teacher elicits questions or comments about the book before reading; teacher 
reads book with expression/displays enjoyment of book), “Book Talk” (e.g., teacher ensures that 
dilemmas/choices in the story are clearly identified and discussed; teacher effectively uses 
discussion to explore the ideas expressed in student role plays), and “Applied Learning” (e.g., 
teacher effectively engages students in skill practice; the connection between the book reading 
and applied learning activities was made to the students). 

Finally, the global quality of teaching will be measured by the CLASS during a single 
live observation in each classroom by an independent observer during the first month of the 
academic year. Observers rate teaching quality on a 7-pt. Likert scale (l=low, 7=high). 

Bivariate correlations will be used to examine the extent to which quality of coaches’ 
online-prompts are related to the quality of in-person conferences between coaches and teachers. 
Independent samples t-tests will be used to assess the extent of and differences in variability for 
each of the two quality measures within coaches, across their caseload of teachers. To examine 
the extent to which quality of coaching predicts variation in teachers’ implementation of the 4Rs 
curriculum, above and beyond a global measure of teaching quality (CLASS), regression 
analyses will be conducted in Mplus (version 5.02, Muthen & Muthen, 2008) using the 
type=complex analysis command to account for the nesting of teachers within coaches. 

Findings / Results: 

Data for the proposed analyses using these measures in the current sample are being collected 
and coded this fall and will be available for analysis by January 2013. Important to note is that 
pilot testing last year of the 2 coaching quality assessment rubrics across 4 coaches and the 4Rs 
implementation quality measure across 12 demographically similar classrooms demonstrated a 
high level of agreement between coders. In addition, 20 videos were coded across these 12 
classrooms for quality of 4Rs implementation, which indicated normal distribution of items and 
variability across classrooms (M= 3.29, 577=. 5 6); this provides confidence that there will be 
variability to predict in this measure across the new sample of 35 classrooms. 

Conclusions: 

We expect to learn how different measures of coaching implementation are associated with one 
another and to identify elements of coaching that may be particularly challenging to standardize. 
Implications will be drawn about how best to provide support to coaches in the field to maintain 
their fidelity to the coaching model and ensure a high quality coaching experience for teachers. 

In addition, we will discuss patterns of prediction from quality of coaching implementation to 
teachers’ fidelity to the 4Rs intervention model in light of implications for identifying elements 
of coaching that might be most strongly linked to teachers’ classroom practice. In particular, we 
will draw attention to the distinction between the on-line element of coaching (prompts) and the 
in-person aspect (conferences), given the significant logistical and resource implications of 
providing support to teachers remotely over the internet versus live in the classroom. 
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